Image capturing apparatus and method for controlling the image capturing apparatus

ABSTRACT

An image capturing apparatus comprises an image sensor and a generation unit configured to generate a signal of each of the pixels from the signals respectively read out from the plurality of charge accumulation units. The image sensor includes a plurality of pixels, each of the pixels including: a photoelectric conversion unit; a plurality of charge accumulation units for accumulating a charge generated in the photoelectric conversion unit; a control unit configured to control accumulation of charge in each of the plurality of charge accumulation units; and a readout unit configured to read out a signal corresponding to the charge from each of the plurality of charge accumulation units. The photoelectric conversion unit is formed in a first chip, and the plurality of charge accumulation units, the control unit, and the readout unit are formed in a second chip.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates to an image capturing apparatus that usesan image sensor typified by a CMOS image sensor, and a method forcontrolling the image capturing apparatus.

Description of the Related Art

An increasing number of recent electronic cameras and video camerasinclude a CMOS image sensing device mounted thereon. A commonly usedCMOS image sensing device employs a scheme in which signal chargesgenerated and accumulated in photoelectric conversion units of pixelsarranged in a two-dimensional matrix are sequentially read out on arow-by-row basis. In recent years, a simultaneous image capturingfunction (global shutter function) of accumulating signal charges at thesame timing has been proposed, and the applications of a CMOS imagesensing element having the global shutter function are also increasing.

According to Japanese Patent Laid-Open No. 2011-217315, a plurality ofCCDs serving as charge holding units arranged in series are provided ineach pixel, and the dynamic range is expanded by obtaining signals withdifferent accumulation periods of time. However, the provision of aplurality of CCDs in a single pixel in a solid-state image capturingapparatus including an increasing number of pixels results in areduction in size of the photodiode portion serving as a photoelectricconversion unit, and thus may reduce the sensitivity.

According to Japanese Patent Laid-Open No. 2011-217315, the output fromthe sensor is performed for one row at a time even though the globalshutter is used, and therefore, it takes a long readout period of timeto obtain signals with different accumulation periods of time.Furthermore, according to Japanese Patent Laid-Open No. 2011-217315, thecharge holding portion is disposed in each of the pixels of a firstchip, and therefore, the size of the photodiode portion is equivalent tothat of the conventional counterpart.

SUMMARY OF THE INVENTION

The present invention has been made in consideration of the abovesituation, and provides an image with a wide dynamic range, withoutreducing the size of the photodiode portion.

According to the present invention, provided is an image capturingapparatus, comprising: an image sensor including a plurality of pixels,each of the pixels including: a photoelectric conversion unit; aplurality of charge accumulation units for accumulating a chargegenerated in the photoelectric conversion unit; a control unitconfigured to control accumulation of charge in each of the plurality ofcharge accumulation units; and a readout unit configured to read out asignal corresponding to the charge from each of the plurality of chargeaccumulation units; and a generation unit configured to generate asignal of each of the pixels from the signals respectively read out fromthe plurality of charge accumulation units, wherein the photoelectricconversion unit is formed in a first chip, and the plurality of chargeaccumulation units, the control unit, and the readout unit are formed ina second chip.

Further, according to the present invention, provided is an imagecapturing apparatus comprising: an image sensor including a plurality ofpixels, each of the pixels including: a photoelectric conversion unit; aplurality of charge accumulation units for accumulating a chargegenerated in the photoelectric conversion unit; a charge holding unitconfigured to hold the charges transferred from the plurality of chargeaccumulation units; a control unit configured to control accumulation ofcharge in each of the plurality of charge accumulation units; a transferunit configured to transfer the charges from the plurality of chargeaccumulation units to the charge holding unit; and a readout unitconfigured to read out each of signals corresponding to the charges heldin the charge holding unit; and a generation unit configured to generatea signal of each of the pixels from the signals read out from the chargeholding unit, wherein the photoelectric conversion unit is formed in afirst chip, and the plurality of charge accumulation units, the chargeholding unit, the control unit, the transfer unit, and the readout unitare formed in a second chip.

Furthermore, according to the present invention, provided is a methodfor controlling an image capturing apparatus including: an image sensorincluding a plurality of pixels, each of the pixels including: aphotoelectric conversion unit and a plurality of charge accumulationunits for accumulating a charge generated in the photoelectricconversion unit, a control unit and a readout unit, wherein thephotoelectric conversion unit is formed in a first chip, and theplurality of charge accumulation units, the control unit, and thereadout unit are formed in a second chip, the method comprising: acharge accumulation step of controlling, by the control unit,accumulation of charge in each of the plurality of charge accumulationunits; a readout step of reading out, by the readout unit, a signalcorresponding to the charge from each of the plurality of chargeaccumulation units; and a generation step of generating, by a generationunit, a signal of each of the pixels from the signals respectively readout from the plurality of charge accumulation units in the readout step.

Further, according to the present invention, provided is a method forcontrolling an image capturing apparatus including: an image sensorincluding a plurality of pixels, each of the pixels including: aphotoelectric conversion unit; a plurality of charge accumulation unitsfor accumulating a charge generated in the photoelectric conversionunit; a charge holding unit configured to hold charges transferred fromthe plurality of charge accumulation units, a control unit, a transferunit, and a readout unit wherein the photoelectric conversion unit isformed in a first chip, and the plurality of charge accumulation units,the charge holding unit, the control unit, the transfer unit, and thereadout unit are formed in a second chip, the method comprising: acontrol step of controlling, by the control unit, accumulation of chargein each of the plurality of charge accumulation units; a transfer stepof transferring, by the transfer unit, the charges from the plurality ofcharge accumulation units to the charge holding unit; a readout step ofreading out, by the readout unit, each of signals corresponding to thecharges held in the charge holding unit; and a generation step ofgenerating, by a generation unit, a signal of each of the pixels fromthe signals respectively read out from the plurality of chargeaccumulation units.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments (with reference to theattached drawings).

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate embodiments of the invention, andtogether with the description, serve to explain the principles of theinvention.

FIG. 1 is a block diagram showing a schematic configuration of an imagecapturing apparatus according to a first embodiment of the presentinvention;

FIG. 2 is a schematic diagram of a stacked structure of an image sensoraccording to the first embodiment;

FIG. 3 is a block diagram showing a configuration of the image sensoraccording to the first embodiment;

FIGS. 4A and 4B are an equivalent circuit diagram and a cross-sectionalview, respectively, of a pixel according to the first embodiment;

FIG. 5 is a timing chart showing a pixel driving pattern according to aconventional technique;

FIG. 6 is a potential transition diagram according to a conventionaltechnique;

FIG. 7 is a timing chart showing a pixel driving pattern according tothe first embodiment;

FIG. 8 is a potential transition diagram according to the firstembodiment;

FIGS. 9A and 9B are graphs illustrating pixel signal levels and outputselection according to the first embodiment;

FIG. 10 is a flowchart illustrating readout control of the image sensoraccording to the first embodiment;

FIG. 11 is a timing chart showing a pixel driving pattern according to asecond embodiment;

FIG. 12 is a potential transition diagram according to the secondembodiment;

FIGS. 13A and 13B are graphs showing pixel signal levels and outputselection according to the second embodiment;

FIG. 14 is a flowchart illustrating readout control of an image sensoraccording to the second embodiment;

FIG. 15 is a block diagram showing a configuration of the image sensoraccording to a third embodiment;

FIGS. 16A and 16B are graphs showing pixel signal levels and outputselection according to the third embodiment;

FIG. 17 is a flowchart illustrating readout control of an image sensoraccording to the third embodiment;

FIG. 18 is a timing chart showing a pixel driving pattern according to afourth embodiment;

FIG. 19 is a potential transition diagram according to the fourthembodiment;

FIG. 20 is a flowchart illustrating readout control of an image sensoraccording to the fourth embodiment;

FIG. 21 is a block diagram showing a configuration of an image sensoraccording to a fifth embodiment;

FIG. 22 is an equivalent circuit diagram of a pixel according to thefifth embodiment;

FIG. 23 is a timing chart showing a pixel driving pattern according tothe fifth embodiment;

FIG. 24 is a potential transition diagram according to the fifthembodiment;

FIG. 25 is a flowchart illustrating readout control of an image sensoraccording to the fifth embodiment;

FIGS. 26A and 26B are diagrams illustrating addition control accordingto a sixth embodiment;

FIG. 27 is a flowchart illustrating readout control of an image sensoraccording to the sixth embodiment; and

FIG. 28 is a block diagram schematically showing a configuration of amobile phone according to a seventh embodiment.

DESCRIPTION OF THE EMBODIMENTS

Exemplary embodiments of the present invention will be described indetail in accordance with the accompanying drawings.

First Embodiment

FIG. 1 is a block diagram showing a configuration of an image capturingapparatus according to a first embodiment of the present invention. Theimage capturing apparatus mainly includes an image processing apparatus100, a recording medium 200 such as a memory card or a hard disk, and alens unit 300.

The lens unit 300 includes a taking lens 310, an aperture 312, a lensmount 316, a lens control unit 320, and a connector 322. The lens mount316 mechanically connects the lens unit 300 to a lens mount 106 of theimage processing apparatus 100. The connector 322 is electricallyconnected to the image processing apparatus 100 via a connector 122 onthe image processing apparatus 100 side. The lens control unit 320receives a signal from the image processing apparatus 100 via theconnectors 322 and the 122, and performs focus adjustment by changingthe position of the taking lens 310 on the optical axis in accordancewith the received signal. The lens control unit 320 receives a signalfrom the image processing apparatus 100 in the same manner, and controlsthe opening of the aperture 312.

In the image processing apparatus 100, an image of incident light madeincident is formed via mirrors 130 and 131 when the mirror 130 is on theoptical axis, and the user can check, from an optical finder 104, theframing of a still image being captured. An image sensor 1400 includescolumn AD circuits and timing control blocks, which will be describedbelow, and photoelectrically converts an optical image made incident viathe lens unit 300 into an electric signal when the mirror 130 iswithdrawn from the optical axis. A shutter 12 controls the amount ofexposure to the image sensor 1400.

An analog front end (AFE) 1700 contains an A/D converter that convertsan analog signal output from the image sensor 1400 into a digitalsignal. A timing generator (TG) 1800 supplies a clock signal and acontrol signal to the image sensor 1400 and the A/D converter of the AFE1700. A system control circuit 50 (hereinafter, referred to as “CPU”)performs overall control of the image processing apparatus 100,including image processing.

A monitor 1200 is constituted by a liquid crystal display (LCD) or thelike, and is capable of displaying a live-view image and a capturedstill image. A shutter switch 61 is used for instructing capturing of astill image, and has a two-stage configuration. In response to anoperation of pressing shallow to the first stage, or the so-calledhalf-pressing operation, automatic focus adjustment and automaticexposure control including setting of the shutter speed and thenumerical aperture by an automatic exposure mechanism in the statebefore capturing are performed. In response to an operation of pressingdeep to the second stage, or the so-called full-pressing operation, theshutter 12 is operated to perform a capturing operation. A moving imagerecording start/stop switch 62 is used for instructing recording of amoving image, and the moving image recording operation is continuouslyperformed after the start of recording has been instructed.

A power switch 60 serves to switch between power-on and power-off of theimage processing apparatus 100. In addition, the setting for power-onand power-off of various accessory devices connected to the imageprocessing apparatus 100, such as the lens unit 300, an external flash,and the recording medium 200 can be switched accordingly.

A volatile memory (RAM) 70 temporarily records image data output fromthe image sensor 1400 and image data that has been subjected to imageprocessing in an image processing unit 72. The RAM 70 also has thefunction of a work memory of the CPU 50. A nonvolatile memory (ROM) 71stores a program used when the CPU 50 performs operations. The imageprocessing unit 72 performs processing such as correction andcompression of a still image. When there are a plurality of outputs fromeach pixel as will be described below, a pixel output selection unit 73selects an output of an appropriate level from these outputs. When thereare a plurality of outputs from each pixel as will be described below, apixel addition unit 74 adds these outputs.

A power control unit 80 is composed of, for example, a battery detectioncircuit, a DC-DC converter, and a switch circuit for switching a blockto which a current is to be applied. Furthermore, the power control unit80 detects the mounting of a battery, the type of the battery, and theremaining battery power, and controls a DC-DC converter based on theresult of detection and an instruction from the CPU 50, and supplies anecessary voltage to various parts including the recording medium for anecessary period. The power control unit 80 is connected to a power unit86 via connectors 82 and 84. The power unit 86 is constituted by aprimary battery such as an alkaline battery or a lithium battery, asecondary battery such as a Li-ion battery, an AC adapter, or the like.

An interface 90 performs communications with the recording medium 200 towhich it is electrically connected via a connector 92. The recordingmedium 200 is a recording medium such as a memory card or a hard disk,and includes a recording unit 252 constituted by a semiconductor memory,a magnetic disk or the like, and a connector 256 to the image processingapparatus 100.

An interface 120 communicates via an electric signal with the lens unit300 to which it is electrically connected via the connector 122. Thelens mount 106 mechanically connects the lens unit 300.

The image sensor 1400 according to the present embodiment has a stackedstructure. FIG. 2 is an oblique projection showing an overview thereof.In the image sensor 1400, a first chip 1401 including a light-receivingportion and a second chip 1402 including a charge holding unit arestacked on each other at the chip level. In the first chip 1401, aportion including a photoelectric conversion unit (photodiode,hereinafter referred to as “PD”) inside a pixel unit 206, which will bedescribed below, is formed. The second chip 1402 is formed by a circuitincluding a charge-coupled device (hereinafter, referred to as “CCD”)serving as a charge holding unit that temporarily holds data transferredfrom the PD.

FIG. 3 is a block diagram showing a configuration of the image sensor1400. Pixels 203 are arranged in a matrix. Hereinafter, a vertical arrayis referred to as “column”, and a horizontal array is referred to as“row”. A pixel unit 206 is a collection of all the columns and rows ofthe pixels 203. A vertical scanning circuit 202 outputs a signalrequired for row selection to read out a selected row and for readout ofcharges of each row to the circuit of each pixel.

The pixels 203 of each column are connected to two vertical output lines421 and 422. The signals output to the vertical output lines 421 and 422are output to horizontal output lines 211 and 212, respectively, viacolumn amplifiers 204 and column circuits 205. Signal outputs for asingle row that are output to the horizontal output lines 211 and 212are sequentially output in the horizontal direction by the driving of ahorizontal scanning circuit (not shown).

FIG. 4A shows an equivalent circuit diagram of a pixel 203. FIG. 4Bshows a cross-sectional view of the pixel 203. In FIG. 4A, a charge thatis generated and accumulated in a PD 408 is temporarily held in CCDs 409and 415 by controlling CCD control signals φ401 and φ404. The chargeheld in the CCD 409 is transferred to a floating diffusion unit(hereinafter, referred to as “FD”) 411 by controlling a transfer switch410 using a transfer control signal φ400. A source follower amplifier413 amplifies the voltage based on the charge accumulated in the FD 411,and outputs the amplified voltage as a pixel signal. The output pixelsignal is output to the vertical output line 421 by controlling a rowselection switch 414 using a row selection control signal φ406. Althoughthe CCDs 409 and 415 are directly connected to the PD 408 in FIG. 4A,another MOS transistor may be connected between the PD 408 and the CCD409 as a transfer gate. Here, the PD 408 is formed in the first chip1401, and the remaining portions are formed in the second chip 1402.

Meanwhile, the charge held in the CCD 415 is transferred to a floatingdiffusion unit (FD) 417 by controlling a transfer switch 416 using atransfer control signal φ403. A source follower amplifier 419 amplifiesthe voltage based on the charge accumulated in the FD 417, and outputsthe amplified voltage as a pixel signal. The output pixel signal isoutput to the vertical output line 422 by controlling a row selectionswitch 420 using a row selection control signal φ407.

To reset unnecessary charges accumulated in the FDs 411 and 417, resetswitches 412 and 418 are controlled using reset control signals φ402 andφ405, respectively. The reset of the PD 408 will be described later.

The CCD control signals φ401 and φ404, the transfer control signals φ400and φ403, the reset control signals φ402 and φ405, and the row selectioncontrol signals φ406 and φ407 are output from the vertical scanningcircuit 202 by being controlled by the CPU 50 via the TG 1800.

FIG. 4B shows a portion of the stacked sensor where each component ofthe second chip 1402 in the equivalent circuit shown in FIG. 4A isdisposed. The charge accumulation portion corresponding to the chargeaccumulated in the PD 408 of the first chip 1401 is a charge holdingunit 408A, and the charge generated in the PD 408 is transferred to thecharge holding unit 408A of the second chip 1402. Here, the boundarybetween the first chip 1401 and the second chip 1402 may be connected bya method using a technique such as microbump. The CCD 409 is disposedadjacent to the charge holding unit 408A. The CCD 409 includes a gateelectrode 409A and a CCD charge holding unit 434, and a transfer switch410A and the FD 411 are disposed adjacent to the CCD 409. Although notshown, the CCD 415 is also disposed adjacent to the charge holding unit408A, as with the CCD 409. The transfer switch 416 and the FD 417 aredisposed in the same manner.

Next, with reference to FIGS. 5 and 6, a description will be given ofthe reset operation of the PD 408 and the charge transfer by theconventional technique that uses only one of the CCDs (here, the CCD409). FIG. 5 is a timing chart of control signals in normal readout.FIG. 6 shows a potential transition diagram in normal readout. Note thatin the potential transition diagrams shown in the following, thepotential for electrons is lower (electric potential is higher) in thedown direction. In addition, the control of the control signals φ400 toφ407 shown in FIG. 5 is performed by the CPU 50 via TG 1800 and thevertical scanning circuit 202.

At timing T500, which is prior to readout, excess charges remain in thecharge holding unit 408A and the CCD 409 (T500 in FIG. 6), andtherefore, these charges need to be reset. Between timings T500 andT501, the transfer control signal φ400, the CCD control signal φ401, andthe reset control signal φ402 are set to HIGH. As a result, thepotential of the CCD 409 is lowered at timing T501, and the potential ofthe transfer switch 410 is also lowered. Thereby, the excess chargesthat have been present in the charge holding unit 408A and the CCD 409are transferred to the CCD 409 and the FD 411 (T501 in FIG. 6). At thistime, the reset control signal φ402 is HIGH, so that the charge in theFD 411 is discharged.

Between timings T501 and T502, the CCD control signal φ401 is set to LOWto raise the potential of the CCD 409. Thereby, all the unnecessarycharges remaining in the CCD 409 are transferred to the FD 411 (T502 inFIG. 6).

Between timings T502 and T503, the transfer control signal φ400 is setto LOW. After all the unnecessary charges have been successfullyremoved, the reset control signal φ402 is also set to LOW (T503 in FIG.6).

Between timings T503 and T504, the accumulation of charge is started.Then, in order to transfer, to the CCD 409, the charge that is generatedin the PD 408 and is transferred to the charge holding unit 408A duringthe charge accumulation period, the CCD control signal φ401 is set toHIGH (T504 in FIG. 6).

After a predetermined charge accumulation period has elapsed since theaccumulation of charge was started, the CCD control signal φ401 is setto LOW immediately before timing T505. At timing T505, the chargegenerated in the PD 408 has been moved to the CCD 409 and is held there(T505 in FIG. 6).

Between timings T505 and T506, the transfer control signal φ400 is setto HIGH. Thereby, at timing T506, the charge held in the CCD 409 istransferred to the FD 411 (T506 in FIG. 6).

Between timings T506 and T507, the transfer control signal φ400 is setto LOW. Thereby, at timing T507, the charges in all pixels have beentransferred to the FDs 411 (T507 in FIG. 6). After timing T507, the rowselection control signal φ406 is set to HIGH on a row-by-row basis, andthereby, the voltage of the FD 411 in which the charge is held isamplified by the source follower amplifier 413, and is output to thevertical output line 421.

Next, with reference to FIGS. 7 and 8, a description will be given ofthe charge transfer by a readout scheme that uses both of the CCDs 409and 415 according to the first embodiment. FIG. 7 shows a timing chartof control signals according to the first embodiment. FIG. 8 shows apotential transition diagram according to the first embodiment. Notethat the control of the control signals φ400 to φ407 shown in FIG. 7 isalso performed by the CPU 50 via the TG 1800 and the vertical scanningcircuit 202.

First, the same control as that performed at timings T500 to T503 inFIGS. 5 and 6 is performed as the reset operation, and therefore, thedetailed description shall be omitted. However, in the readout schemethat uses both of the CCDs 409 and 415 according to the firstembodiment, the control signals φ403 to φ405 are controlled in the samemanner as the control signals φ400 to φ402. After resetting, at timingT700, all the excess charges in the charge holding unit 408A, the CCD409, the CCD 415, the FD 411, and the FD 417 have been reset (T700 inFIG. 8).

Between timings T700 and T701, the accumulation of charge is started.The CCD control signals φ401 and φ404 are set to HIGH, and thereby, attiming T701, the charge generated in the PD 408 during the chargeaccumulation period is transferred to each of the CCDs 409 and 415 viathe charge holding unit 408A (T701 in FIG. 8).

After a predetermined charge accumulation period has elapsed since theaccumulation of charge was started, the CCD control signal φ401 is setto LOW immediately before timing T702, and the potential of the CCD 409is raised at timing T702 (T702 in FIG. 8). Accordingly, from timing T702and onward, the charge generated in the PD 408 is transferred only tothe CCD 415 (T702 to T704 in FIG. 8).

Between timings T702 and T703, the transfer control signal φ400 is setto HIGH. Thereby, at timing T703, the charge that has been held in theCCD 409 is transferred to the FD 411 (T703 in FIG. 8).

Between timings T703 and T704, the transfer control signal φ400 is setto LOW. Thereby, at timing T704, the charges in all pixels have beentransferred to the FDs 411 (T704 in FIG. 8). After timing T704, the rowselection control signal φ406 is set to HIGH on a row-by-row basis, andthereby the voltage of the FD 411 in which the charge is held isamplified by the source follower amplifier 413, and is output to thevertical output line 421.

On the other hand, the CCD control signal φ404 is set to LOW immediatelybefore timing T705. Thereby, at timing T705, the potential of the CCD415 is raised, and the transfer from the charge holding unit 408A to theCCD 415 is stopped (T705 in FIG. 8).

Between timings T705 and T706, the transfer control signal φ403 is setto HIGH. Thereby, at timing T706, the charge that has been held in theCCD 415 is transferred to the FD 417 (T706 in FIG. 8).

Between timings T706 and T707, the transfer control signal φ403 is setto LOW. Thereby, at timing T707, the charges in all pixels have beentransferred to the FDs 417 (T707 in FIG. 8). After timing T707, the rowselection control signal φ407 is set to HIGH on a row-by-row basis, andthereby the voltage of the FD 417 in which the charge is held isamplified by the source follower amplifier 419, and is output to thevertical output line 422.

FIGS. 9A and 9B are graphs illustrating the levels of pixel signals ofspecific pixels and the output selection therefor according to the firstembodiment, showing the levels of the pixel signals of specific pixels.L900 denotes the saturation level. FIG. 9A shows the output obtained bythe driving of the CCD 409 side of the configuration described withreference to FIG. 8, or in other words, the output obtained by thedriving with a short accumulation time. The output of a pixel P902 is alevel L905, which is considerably small relative to the saturation levelL900. The output of a pixel P903 is a level L904. In contrast, FIG. 9Bshows the output obtained by the CCD 415 side of the configurationdescribed with reference to FIG. 8, or in other words, the outputobtained by the driving with a long accumulation time. The output of thepixel P902 is a level L907, and the output of the pixel P903 reaches thesaturation level L900.

Based on this result, the CPU 50 selects, via the pixel output selectionunit 73, the level L907, which is output by using the CCD 415, from thepixel P902 and selects the level L904, which is output by using the CCD409, from the pixel P903. Note that the level conversion is performed asneeded for each output in order to maintain the consistency of output.For example, it is assumed that the accumulation period of time for thesignal of the CCD 415 is twice as long as that for the signal of the CCD409. Under this condition, the CPU 50 applies a gain of ½ to the outputof the CCD 415 via the image processing unit 72. This output selectionfor outputting an appropriate level is performed for all pixels.

FIG. 10 is a flowchart illustrating readout control of the image sensor1400 according to the first embodiment. When the image capturing starts,the setting is checked to determine whether the mode is to be set to thedynamic range (D-range) expansion mode (S100). If it is not to be set tothe D-range expansion mode, the charge is transferred from the PD 408only to the CCD 409 (S112), and is independently output from all pixelsby normal output (S113), as described with reference to FIGS. 5 and 6.

If it is to be set to the D-range expansion mode, the CPU 50 controls tolower the potentials of the CCDs 409 and 415 via the TG 1800 and thevertical scanning circuit 202 so that the charges are transferred to theCCD 409 and the CCD 415 (S101). After a predetermined chargeaccumulation period for the CCD 409 has elapsed (YES at S102), the CPU50 controls to restore the potential of the CCD 409 via the TG 1800 andthe vertical scanning circuit 202 (S103). Then, the CPU 50 controls toread out the accumulated charge from the CCD 409 via the TG 1800 and thevertical scanning circuit 202 (S104).

After the accumulation period for the other CCD 415 has elapsed (YES atS105), the CPU 50 controls to restore the potential of the CCD 415 viathe TG 1800 and the vertical scanning circuit 202 (S106). Then, the CPU50 controls to read out the accumulated charge from the CCD 415 via theTG 1800 and the vertical scanning circuit 202 (S107).

When the signals read out from the CCDs 409 and 415 are all present inthe memory of the CPU 50, it is checked whether the output of the CCD415 for a single pixel read out with a longer time is the saturationlevel (S108). If it is saturated, the output of the CCD 409 is selected(S109). If it is not saturated, the output of the CCD 415 is selected(S110). The level comparison and the output selection at this time areperformed by the pixel output selection unit 73 under control of the CPU50.

It is checked whether the above-described checking of the pixel outputhas ended for all pixels (S111). If it has not ended, the processreturns to S108, at which the checking is performed for the next pixel.When the checking has ended for all pixels, the readout ends.

As described above, according to the first embodiment, a plurality ofCCD portions are connected to a PD, and outputs with differentaccumulation periods of time are respectively obtained by the CCDportions. Thereby, it is possible to generate an image of an appropriateoutput level in each pixel.

Second Embodiment

Next, a second embodiment of the present invention will be described.Note that the configurations of the image capturing apparatus and theimage sensor according to the second embodiment are the same as thosedescribed with reference to FIGS. 1 to 4B in the first embodimentdescribed above, and therefore, the description thereof shall be omittedhere. In the second embodiment, each of the CCDs 409 and 415 is drivenat the same timing such that the charge accumulation period is the same,and whether to add or independently output the two outputs in a laterstage is selected.

With reference to FIGS. 11 and 12, a description will be given of chargetransfer by a readout scheme according to the second embodiment. FIG. 11shows a timing chart of control signals according to the secondembodiment. FIG. 12 shows a potential transition diagram according tothe second embodiment. Note that the control of the control signals φ400to φ407 shown in FIG. 11 is performed by the CPU 50 via the TG 1800 andthe vertical scanning circuit 202.

First, as with the first embodiment, the same operations as thoseperformed at timings T500 to T503 in FIGS. 5 and 6 are performed as thereset operations, and therefore, the detailed description thereof shallbe omitted here. However, the readout scheme according to the secondembodiment uses both of the CCDs 409 and 415, and therefore, the controlsignals φ403 to φ405 are controlled in the same manner as the controlsignals φ400 to φ402. After resetting, at timing T1100, all the excesscharges in the charge holding unit 408A, the CCD 409, the CCD 415, theFD 411, and the FD 417 have been reset (T1100 in FIG. 12).

Between timings T1100 and T1101, the accumulation of charge is started.Then, the CCD control signals φ401 and φ404 are set to HIGH, andthereby, the charge generated in the PD 408 during the chargeaccumulation period is transferred to each of the CCD 409 and the CCD415 via the charge holding unit 408A at timing T1101 (T1101 in FIG. 12).

After a predetermined charge accumulation period has elapsed since theaccumulation of charge was started, the CCD control signals φ401 andφ404 are set to LOW immediately before timing T1102, thereby raising thepotentials of the CCDs 409 and 415 at timing T1102 (T1102 in FIG. 12).

Between timings T1102 and T1103, the transfer control signals φ400 andφ403 are set to HIGH. Thereby, at timing T1103, the charge that has beenheld in the CCD 409 is transferred to the FD 411, and the charge thathas been held in the CCD 415 is transferred to the FD 417 (T1103 in FIG.12).

Between timings T1103 and T1104, the transfer control signals φ400 andφ403 are set to LOW. Thereby, at timing T1104, the charges in all pixelshave been transferred to the FD 411 and the FD 417 (T1104 in FIG. 12).After timing T1104, each of the row selection control signals φ406 andφ407 is set to HIGH on a row-by-row basis, and thereby, the voltage ofthe FD 411 in which the charge is held is amplified by the sourcefollower amplifier 413, and is output to the vertical output line 421.Additionally, the voltage of the FD 417 is amplified by the sourcefollower amplifier 419, and is output to the vertical output line 422.

FIGS. 13A and 13B are graphs illustrating the levels of pixel signals ofspecific pixels and the output selection therefor according to thesecond embodiment. As with FIGS. 9A and 9B, the levels of the pixelsignals of specific pixels are shown. L900 is the saturation level. FIG.13A shows the output of a single CCD. Here, for example, the output ofthe CCD 409 is shown. The output of the pixel P902 is a level L1302,which is considerably small relative to the saturation level L900. Theoutput of the pixel P903 is a level L1301. FIG. 13B shows the levelswhen the outputs of the two CCDs 409 and 415 are added. The added outputof the pixel P902 is a level L1304. On the other hand, the added outputof the pixel P903 reaches the saturation level L900.

Based on this result, the CPU 50 selects the level L1304, which isobtained by adding the outputs of the CCD 409 and the CCD 415, as theoutput of the pixel P902, and selects the level L1301, which is theoutput of the CCD 409, as the output of the pixel P903, via the pixeladdition unit 74. Note that the level conversion is performed as neededfor each output in order to maintain the consistency of output. Forexample, it is assumed that the level obtained when addition isperformed is about twice the level obtained when addition is notperformed. Under this condition, the CPU 50 applies, via the imageprocessing unit 72, a gain of ½ times to the output resulting fromaddition.

FIG. 14 is a flowchart illustrating readout control of the image sensor1400 according to the second embodiment. When the image capturingstarts, the setting is checked to determine whether the mode is to beset to the D-range expansion mode (S200). If it is not to be set to theD-range expansion mode, the charge is transferred from the PD 408 onlyto the CCD 409 (S209), and is independently output from all pixels bynormal output (S210), as described with reference to FIGS. 5 and 6 inthe first embodiment.

If it is to be set to the D-range expansion mode, the CPU 50 controls tolower the potentials of the CCDs 409 and 415 via the TG 1800 and thevertical scanning circuit 202 so that the charges are transferred to theCCDs 409 and 415 (S201). Then, after a predetermined charge accumulationperiod has elapsed (YES at S202), the CPU 50 controls to restore thepotentials of the CCDs 409 and 415 via the TG 1800 and the verticalscanning circuit 202 (S203). Then, the CPU 50 controls to read out theaccumulated charges from the CCDs 409 and 415 via the TG 1800 and thevertical scanning circuit 202 (S204).

When the data read out from the CCDs 409 and 415 is all present in thememory of the CPU 50, it is checked whether the level obtained when thetwo signals are added is saturated (S205). If it is not saturated, theresult of adding the signals of the CCDs 409 and 415 is selected (S207).If it is saturated, the output of the CCD 409 is selected (S206). Notethat the output obtained by averaging the levels of the CCDs 409 and 415may be selected.

Then, it is checked whether the above-described checking of the pixeloutput has ended for all pixels (S208). If it has not ended, the processreturns to S205, at which the checking is performed for the next pixel.When the checking has ended for all pixels, the readout ends.

As described above, according to the second embodiment, a plurality ofCCD portions are connected to a PD, and the outputs of the CCD portionsdriven at at the same timing and having the same charge accumulationperiod are obtained. By selecting whether or not to add the outputs, itis possible to generate an image of an appropriate output level in eachpixel.

Third Embodiment

Next, a third embodiment of the present invention will be described.Note that the configurations of the image capturing apparatus and theimage sensor according to the third embodiment are different in theconfiguration of the image sensor 1400 from those described withreference to FIGS. 1 to 4B in the first embodiment described above. Therest of the configurations are the same, and therefore, the descriptionthereof shall be omitted here. Additionally, in the third embodiment, aswith the second embodiment, each of the CCDs 409 and 415 is driven atthe same timing as shown in FIG. 11 such that the charge accumulationperiod is the same. Different gains are applied to the column amplifierswhen the charge is readout from each of the CCDs 409 and 415.

FIG. 15 is a block diagram showing a configuration of an image sensoraccording to the third embodiment. Regarding the pixels, the columnamplifiers 204, the column circuits 205 and the like, the columnamplifier and the column circuit connected to the column 1502 of thepixel unit 206 included in the configuration of the image sensor 1400shown in FIG. 3 are shown. Here, the column amplifier and the columncircuit that are connected to the CCD 409 are denoted by 204 and 205,respectively, and the column amplifier and the column circuit that areconnected to the CCD 415 are denoted by 1501 and 1500, respectively.

In the second embodiment described above, after the signals read outfrom the CCDs 409 and 415 have been output to the vertical output lines421 and 422, the same gain is applied to the column amplifiers 204. Incontrast, in the third embodiment, different gains are applied to thecolumn amplifiers 204 and 1501.

FIGS. 16A and 16B are graphs illustrating the levels of pixel signals ofspecific pixels and the output selection therefor according to the thirdembodiment. As with FIGS. 9A and 9B, the levels of pixel signals ofspecific pixels are shown. L900 is the saturation level. FIG. 16A showsthe output of the CCD 409. The output levels L1301 and L1302 of thepixels P902 and P903 are the same as those shown in FIG. 13A. FIG. 16Bshows the output of the CCD 415. Here, an example is shown in which thecolumn amplifier 1501 connected to the CCD 415 has a larger gain thanthe column amplifier 204 connected to the CCD 409. This shows that theoutput level of the pixel P902 is L1601, and the output level of thepixel P903 is the saturation level L900.

Based on this result, the CPU 50 controls to select the output levelL1601 of the CCD 415 as the output of the pixel P902 via the pixeloutput selection unit 73. As the output of the pixel P903, the CPU 50controls to select the output level L1301 of the CCD 409. Note that thelevel conversion is performed as needed for each output in order tomaintain the consistency of output. For example, the CPU 50 multipliesthe output level of the CCD 415 by a gain ratio G2/G1 of the columnamplifiers in the image processing unit 72, for example, when G1represents the gain of the column amplifier 1501 and G2 represent thegain of the column amplifier 204, thus maintaining the consistency.

FIG. 17 is a flowchart illustrating readout control of the image sensor1400 according to the third embodiment. Note that the processing of S200to S203, S209 and S210 are the same as the processing described withreference to FIG. 14 in the second embodiment, and therefore, thedescription thereof shall be omitted here.

After a predetermined charge accumulation period of time has elapsed,the CPU 50 controls to read out the accumulated charges from the CCD 409and the CCD 415 via the TG 1800 and the vertical scanning circuit 202.At this time, the gain of the column amplifier 1501 connected to the CCD415 is larger than the gain of the column amplifier 204 connected to theCCD 409 (S304).

When the data read out from the CCD 409 and the CCD 415 is all presentin the memory of the CPU 50, it is checked whether the level of thesignal read out from the CCD 415 is saturated (S305). If it is notsaturated, the output of the CCD 415 to which the column amplifier 1501having a larger gain is connected is selected (S307). If it issaturated, the output of the CCD 409 is selected (S306).

Then, it is checked whether the above-described checking of the pixeloutput has ended for all pixels (S308). If it has not ended, the processreturns to S305, at which the checking is performed for the next pixel.When the checking has ended for all pixels, the readout ends.

As described above, according to the third embodiment, a plurality ofCCD portions are connected to a PD, different gains are applied by thecolumn amplifiers to the outputs of the CCD portions driven at the sametiming and having the same charge accumulation period, and one of theoutputs is selected. This makes it possible to generate an image of anappropriate output level in each pixel.

Fourth Embodiment

Next, a fourth embodiment of the present invention will be described.Note that the configurations of the image capturing apparatus and theimage sensor according to the fourth embodiment are the same as thosedescribed with reference to FIGS. 1 to 4B in the first embodimentdescribed above, and therefore, the description thereof shall be omittedhere. In the fourth embodiment, the CCDs 409 and 415 are driven atdifferent timings for the same period, and whether to add the outputs ofthe CCDs 409 and 415 or to select and output a portion thereof isselected.

With reference to FIGS. 18 and 19, a description will be given of chargetransfer by a readout scheme according to the fourth embodiment. FIG. 18shows a timing chart of control signals according to the fourthembodiment. FIG. 19 shows a potential transition diagram according tothe fourth embodiment. Note that the control of the control signals φ400to φ407 shown in FIG. 18 is performed by the CPU 50 via the TG 1800 andthe vertical scanning circuit 202.

First, as with the first embodiment, the same operations performed attimings T500 to T503 shown in FIGS. 5 and 6 are performed as the resetoperations, and therefore, the detailed description thereof shall beomitted here. However, the readout scheme of the fourth embodiment usesboth of the CCDs 409 and 415, and therefore, the control signals φ403 toφ405 are controlled in the same manner as the control signals φ400 toφ402. After resetting, all the excess charges in the charge holding unit408A, the CCD 409, the CCD 415, the FD 411, and the FD 417 have beenreset at timing T1800 (T1800 in FIG. 19).

Between timings T1800 and T1801, the accumulation of charge is started.Then, the CCD control signal φ401 is set to HIGH, and thereby, thecharge generated in the PD 408 during the charge accumulation period istransferred to the CCD 409 via the charge holding unit 408A at timingT1801 (T1801 in FIG. 19).

After a predetermined charge accumulation period has elapsed since theaccumulation of charge was started, the CCD control signal φ401 is setto LOW immediately before timing T1802, and thereby, the potential ofthe CCD 409 is raised at timing T1802 (T1802 in FIG. 19).

Between timings T1802 and T1803, the transfer control signal φ400 is setto HIGH. Thereby, at timing T1803, the charge that has been held in theCCD 409 is transferred to the FD 411 (T1803 in FIG. 19).

Between timings T1803 and 1804, the transfer control signal φ400 is setto LOW, and thereafter, the CCD control signal φ404 is set to HIGH.Thereby, at timing T1804, the charges in all pixels have beentransferred to the FDs 411. After timing T1804, the row selectioncontrol signal φ406 is set to HIGH on a row-by-row basis, and thereby,the voltage of the FD 411 in which the charge is held is amplified bythe source follower amplifier 413, and is output to the vertical outputline 421. Additionally, at T1804, the potential of the CCD 415 islowered, and thereby, the charge generated in the PD 408 during thecharge accumulation period is transferred to the CCD 415 (T1804 in FIG.19).

After a predetermined charge accumulation period has elapsed since theaccumulation of charge in the CCD 415 was started, the CCD controlsignal φ404 is set to LOW immediately before timing T1805 to raise thepotential of the CCD 415 at timing T1805. Thereby, the transfer from thecharge holding unit 408A to the CCD 415 is stopped (T1805 in FIG. 19).

Between timings T1805 and T1806, the transfer control signal φ403 is setto HIGH. Thereby, at timing T1806, the charge that has been held in theCCD 415 is transferred to the FD 417 (T1806 in FIG. 19).

Between timings T1806 and T1807, the transfer control signal φ403 is setto LOW. Thereby, at timing T1807, the charges in all pixels have beentransferred to the FDs 417 (T1807 in FIG. 19). After timing T1807, therow selection control signal φ407 is set to HIGH on a row-by-row basis,and thereby, the voltage of the FD 417 in which the charge is held isamplified by the source follower amplifier 419, and is output to thevertical output line 422.

According to the fourth embodiment, whether to select one of the outputsor to add the outputs can be decided based on the result of continuousaccumulation. For a pixel whose outputs from the CCDs 409 and 415 areboth significantly lower than a predetermined level, the outputs may beadded in the same manner as in the second embodiment. If the addition isnot performed, either of the outputs may be selected. For example, whenthere are three CCDs and accumulation of charge is performed at separatetimings, the output of the CCD for which the accumulation is performedsecond, which corresponds to the timing close to the middle of thecharge accumulation, may always be selected, or the output of the CCDfor which the accumulation is performed first, which corresponds to thetiming close to the start of the charge accumulation, may always beselected. Alternatively, based on the information on camera shake, theoutput obtained during the accumulation period that is less affected bycamera shake may be selected. Note that in the fourth embodiment aswell, the level conversion is performed as needed for each output inorder to maintain the consistency of output, as with the first to thirdembodiments.

FIG. 20 is a flowchart illustrating readout control of the image sensor1400 according to the fourth embodiment. When the image capturingstarts, the setting is checked to determine whether the mode is to beset to the D-range expansion mode (S400). If it is not to be set to theD-range expansion mode, the charge is transferred from the PD 408 onlyto the CCD 409 (S413), and is independently output from all pixels bynormal output (S414), as described with reference to FIGS. 5 and 6.

If it is to be set to the D-range expansion mode, the CPU 50 firstcontrols to lower the potential of the CCD 409 via the TG 1800 and thevertical scanning circuit 202 such that the charge is transferred to theCCD 409 (S401). After a predetermined charge accumulation period for theCCD 409 has elapsed (YES at S402), the CPU 50 controls to restore thepotential of the CCD 409 via the TG 1800 and the vertical scanningcircuit 202 (S403). Then, the CPU 50 controls to read out theaccumulated charge from the CCD 409 via the TG 1800 and the verticalscanning circuit 202 (S404).

In parallel with this, the CPU 50 controls to lower the potential of theCCD 415 via the TG 1800 and the vertical scanning circuit 202 such thatcharge is transferred to the CCD 415 (S405). After a predeterminedcharge accumulation period for the CCD 415 has elapsed (YES at S406),the CPU 50 controls to restore the potential of the CCD 415 via the TG1800 and the vertical scanning circuit 202 (S407). Then, the CPU 50controls to read out the accumulated charge from the CCD 415 via the TG1800 and the vertical scanning circuit 202 (S408).

When the signals read out from the CCDs 409 and 415 are all present inthe memory of the CPU 50, it is checked, for the signals of the samepixel, whether the level of a signal obtained by adding the two signalsread out from the CCDs 409 and 415 is the saturation level (S409). If itis not saturated, the CPU 50 controls to add the two signals via thepixel addition unit 74 (S410). If it is saturated, the CPU 50 selectsone of the outputs of the CCDs 409 and 415 (S411).

It is checked whether the above-described checking of the pixel outputhas ended for all pixels (S412). If it has not ended, the processreturns to S409, at which the checking is performed for the next pixel.When the checking has ended for all pixels, the readout ends.

As described above, according to the fourth embodiment, a plurality ofCCD portions are connected to a PD, and accumulation of charge iscontinuously performed at separate timings. Outputs for which theaccumulation timings are different are obtained, and whether or not toadd the outputs can be selected according to the levels of the obtainedsignals. Thereby, it is possible to generate an image of an appropriateoutput level in each pixel.

Fifth Embodiment

Next, a fifth embodiment of the present invention will be described. Theimage sensor 1400 according to the fifth embodiment is different fromthose of the first to fourth embodiments in that a single FD isconnected to two CCDs, whereas an FD is connected to each of the twoCCDs in the first to fourth embodiments. Note that the rest of theconfiguration of the image capturing apparatus are the same as those ofthe configuration described with reference to FIGS. 1 and 2 in the firstembodiment, and therefore, the description thereof shall be omittedhere.

FIG. 21 is a block diagram showing a configuration of the image sensor1400 according to the fifth embodiment. The difference with theconfiguration shown in FIG. 3 is that two sets of column amplifiers andcolumn circuits provided for each column are replaced by a single set ofa column amplifier and a column circuit.

FIG. 22 is an equivalent circuit diagram of a pixel 203 according to thefifth embodiment. The difference with the configuration shown in FIGS.4A and 4B is that the FD 411 connected to the CCD 409 is also used asthe FD connected to a later stage of the CCD 415 via the transfer switch416.

Next, with reference to FIGS. 23 and 24, a description will be given ofcharge transfer by a readout scheme that uses both of the CCDs 409 and415 according to the fifth embodiment. FIG. 23 shows a timing chart ofcontrol signals according to the fifth embodiment. FIG. 24 shows apotential transition diagram according to the fifth embodiment. Notethat the control of the control signals φ400 to φ406 shown in FIG. 23 isperformed by the CPU 50 via the TG 1800 and the vertical scanningcircuit 202.

First, as with the first embodiment, the same operations as thoseperformed at timings T500 to T503 in FIGS. 5 and 6 are performed as thereset operations, and therefore, the detailed description thereof shallbe omitted here. However, the readout scheme according to the fifthembodiment uses both of the CCDs 409 and 415, and therefore, the controlsignals φ403 and φ404 are controlled in the same manner as the controlsignals φ400 and φ401. After resetting, at timing T2400, all the excesscharges in the charge holding unit 408A, the CCD 409, the CCD 415, theFD 411, and the FD 417 have been reset (T2400 in FIG. 24).

Between timings T2400 and T2401, the accumulation of charge is started.Then, the CCD control signal φ401 and the CCD control signal φ404 areset to HIGH. At timing T2401, the charge generated in the PD 408 duringthe charge accumulation period is transferred to each of the CCD 409 andthe CCD 415 via the charge holding unit 408A (T2401 in FIG. 24).

After a predetermined charge accumulation period has elapsed since theaccumulation of charge was started, the CCD control signal φ401 and theCCD control signal φ404 are set to LOW immediately before timing T2402.Thereby, at timing T2402, the potentials of the CCD 409 and the CCD 415are raised (T2402 in FIG. 24).

Between timings T2402 and T2403, the transfer control signal φ400 is setto HIGH. Thereby, at timing T2403, the charge that has been held in theCCD 409 is transferred to the FD 411 (T2403 in FIG. 24).

Thereafter, between timings T2403 and T2404, the transfer control signalφ400 is set to LOW. Thereby, at timing T2404, the charges in all pixelshave been transferred to the FDs 411. After timing T2404, the rowselection control signal φ406 is set to HIGH on a row-by-row basis, andthereby, the voltage of the FD 411 in which the charge is held isamplified by the source follower amplifier 413, and is output to thevertical output line 421 (T2404 in FIG. 24). At timing T2405, thetransfer to the vertical line of the signals corresponding to thecharges transferred to the FD 411 ends for all rows.

Between timings T2405 and T2406, the transfer control signal φ403 is setto HIGH. Thereby, at timing T2406, the charge that has been held in theCCD 409 is transferred to the FD 411 so as to be added to the chargethat was originally transferred from the CCD 409 (T2406 in FIG. 24).

Thereafter, between timings T2406 and T2407, the transfer control signalφ403 is set to LOW. Thereby, at timing T2407, the charges in all pixelshave been transferred to the FDs 411. After timing T2407, the rowselection control signal φ406 is set to HIGH on a row-by-row basis, andthereby, the voltage of the FD 411 in which the charge is held isamplified by the source follower amplifier 419, and is output to thevertical output line 421 (T2407 in FIG. 24). Note that in the fifthembodiment as well, the level conversion is performed as needed for eachoutput in order to maintain the consistency of output, as with the firstto fourth embodiments.

FIG. 25 is a flowchart illustrating readout control of the image sensor1400 according to the fifth embodiment. When the image capturing starts,the setting is checked to determine whether the mode is to be set to theD-range expansion mode (S500). If it is not to be set to the D-rangeexpansion mode, the charge is transferred from the PD 408 only to theCCD 409 (S513), and is independently output from all pixels by normaloutput (S514), as described with reference to FIGS. 5 and 6.

If it is to be set to the D-range expansion mode, the CPU 50 controls tolower the potentials of the CCDs 409 and 415 via the TG 1800 and thevertical scanning circuit 202 so as to transfer the signals to the twoCCDs 409 and 415 such that the charges are transferred to the CCD 409and the CCD 415 (S501). Then, after a predetermined charge accumulationperiod has elapsed (YES at S502), the CPU 50 controls to restore thepotentials of the CCDs 409 and 415 via the TG 1800 and the verticalscanning circuit 202 (S503). Then, the CPU 50 controls to raise thepotential of the transfer switch 410 via the TG 1800 and the verticalscanning circuit 202 to transfer only the charge accumulated in the CCD409 to the FD 411, and to read out the charge in the CCD 409 via thevertical scanning circuit 202 (S504).

When the signal of the CCD 409 is read out from the FDs 411 of all rowsto the vertical output line 421 (S505), the CPU 50 controls to transferthe charge accumulated in the CCD 415 to the FD 411 via the TG 1800 andthe vertical scanning circuit 202. Thereby, the charge accumulated inthe CCD 409 and the charge accumulated in the CCD 415 are added in theFD 411. Then, the CPU 50 controls to read out the charge accumulated inthe FD 411 via the TG 1800 and the vertical scanning circuit 202 (S506).

When a single signal obtained by non-additional readout from the CCD 409and a signal obtained by additional readout from the CCD 409 and the CCD415 are all present in the memory of the CPU 50 (S507), it is checkedwhether the level of the signal obtained by additional readout is thesaturation level (S509). If it is not saturated, the signal obtained byadditional readout is selected (S510). If it is saturated, the signalread out from the CCD 409 is selected (S511). The above processes arerepeated from S509 until it is determined in S512 that selection for allpixels are completed.

As described above, according to the fifth embodiment, a plurality ofCCD portions are connected to a PD. When the CCDs are connected to asingle FD, a single signal obtained by non-additional readout and asignal obtained by addition in the FD are sequentially read out, and oneof the signals is selected according to the signal level. Accordingly,it is possible to obtain an added signal with a lower noise although ittakes longer to perform readout compared with the configuration of thesecond embodiment since there is only one vertical line.

Sixth Embodiment

Next, a sixth embodiment according to the present invention will bedescribed. According to the sixth embodiment, when a moving image isrecorded in the example shown in the fifth embodiment in which pixels ofa low signal level are added, whether to individually transfer signalsread out in a plurality of CCDs without adding the signals, or totransfer signals after adding them is determined based on the signal ofthe previous frame. Note that the configurations of the image capturingapparatus and the image sensor are the same as those described withreference to FIGS. 1, 2, 21, and 22 in the first and fifth embodiments,and therefore, the description thereof shall be omitted here.

FIGS. 26A and 26B show frames and pixel levels at the time of recordinga moving image according to the sixth embodiment. FIG. 26A shows atransition over time of frames read out in a predetermined cycle, andshows how signals are read out, for example, every 1/30 seconds, firstfrom a frame f2600, and then from frames f2601, f2602, and f2603. FIG.26B shows the transition of the signal levels of pixels pt2604, pt2605,pt2606, and pt2607 located at the same position of the respective framesat that time. In FIG. 26B, a level L900 is the saturation level, and alevel L950 is a first threshold level used for a determination describedbelow, and is, for example, a level that is one half the saturationlevel. A level L960 is a second threshold level used for a determinationdescribed below, and is, for example, a level that is about one half thefirst threshold level L950.

It is assumed that the signal of the pixel pt2604 is obtained by normalreadout using a single CCD 409. In the example shown in FIG. 26B, thelevel is gradually lowered in the order of the level L2608 of the pixelpt2604, the level L2609 of the pixel pt2605, the level L2610 of thepixel pt2606, and so forth. Since the level of the pixel pt2607 isexpected to be lower than L2610, a signal obtained by adding the chargesin the two CCDs 409 and 415 is output. Accordingly, the level of thepixel pt2607 is a level L2612 as a result of the addition although itwould have been L2611 by normal readout.

According to the sixth embodiment, the above-described transition ofsignals for each frame in a moving image is checked, and which of single(non-additional) readout and additional readout is performed for readingout the next frame is determined for each row. FIG. 27 is a flowchartillustrating readout control of the image sensor 1400 according to thesixth embodiment. As described with reference to FIG. 25 in the fifthembodiment, after the accumulation of charge in the two CCDs 409 and 415has ended by the processing up to S503, the processing of determiningwhether to perform single readout or additional readout for each row isperformed at S600. Here, it is determined whether a pixel whose signallevel obtained by normal readout using a single CCD 409 is higher thanthe first threshold is present in the rows of the previous frame orwhether a pixel having a signal level lower than the second threshold isnot present in the rows of the previous frame. If YES at S600,additional readout will result in a saturated pixel, or the signal levelis sufficient without performing additional readout. Therefore, theprocess proceeds to S504, at which single readout is performed. If NO atS600, or in other words, a pixel having a signal level lower than thesecond threshold level L960 is present and a pixel having a signal levelhigher than the first threshold level L950 is not present, the processproceeds to S601, at which the charges are transferred from the two CCDs409 and 415 to the FD 411, and are read out. The above processes arerepeated from S600 until it is determined in S602 that all rows are readout.

As described above, according to the sixth embodiment, switching betweensingle readout and additional readout is performed for each row, andthereby, only a single readout is necessary for each row. Thus, it ispossible to obtain an image of an appropriate output level with a highdegree of speed and accuracy. When an appropriate image cannot beobtained by performing setting for each row, the signal level of theentire frame may be checked, and the entire readout may be performed byeither single readout or additional readout.

Seventh Embodiment

FIG. 28 is a block diagram showing an arrangement of a cellular phone500 according to the seventh embodiment of the present invention. Thecellular phone 500 according to the seventh embodiment has the e-mailfunction, Internet connection function, image shooting/playbackfunction, and the like, in addition to the voice communication function.

In FIG. 28, a communication unit 501 communicates voice data and imagedata with another phone by a communication method complying with acommunication carrier with which the user has a contract. In voicecommunication, a voice processing unit 502 converts voice data from amicrophone 503 into a format suited for origination, and sends theconverted data to the communication unit 501. Also, the voice processingunit 502 decodes voice data from a call destination that has been sentfrom the communication unit 501, and sends the decoded data to aloudspeaker 504.

An image sensing unit 505 includes the image sensor 1400 described inone of the first and sixth embodiments, shoots an image of an object,and outputs image data. At the time of shooting an image, an imageprocessing unit 506 processes image data shot by the image sensing unit505, converts the data into a format suited for recording, and outputsthe converted data. At the time of playing back a recorded image, theimage processing unit 506 processes the image to be played back, andsends the processed image to a display unit 507. The display unit 507includes a liquid crystal display panel about several inches in size,and displays various display contents in accordance with instructionsfrom a control unit 509. A nonvolatile memory 508 stores information ofan address book, and data such as data of e-mail and image data shot bythe image sensing unit 505.

The control unit 509 includes a CPU and memory, and controls therespective units of the cellular phone 500 in accordance with a controlprogram stored in the memory (not shown). An operation unit 510 includesa power button, number keys, and various other operation keys used toinput data by the user. A card I/F 511 records and reads out variousdata on and from a memory card 512. An external I/F 513 transmits, to anexternal device, data stored in the nonvolatile memory 508 and memorycard 512, and receives data transmitted from an external device. Theexternal I/F 513 performs communication by a well-known communicationmethod such as wireless communication or a wired communication methodconforming to, for example, a USB standard.

Next, the voice communication function in the cellular phone 500 will beexplained. When calling a call destination, the user operates the numberkeys of the operation unit 510 to input the number of the calldestination, or operates to display, on the display unit 507, an addressbook stored in the nonvolatile memory 508, selects the call destination,and instructs origination. When the origination is instructed, thecontrol unit 509 originates a call to the call destination via thecommunication unit 501. If the call destination answers the call, thecommunication unit 501 outputs voice data of the destination to thevoice processing unit 502, and also transmits voice data of the user tothe destination.

When transmitting an e-mail, the user instructs creation of a mail byusing the operation unit 510. When the creation of a mail is instructed,the control unit 509 displays a mail creation screen on the display unit507. The user inputs a transmission destination address and body byusing the operation unit 510, and instructs transmission. Whentransmission of the mail is instructed, the control unit 509 sendsaddress information and data of the mail body to the communication unit501. The communication unit 501 converts the mail data into a formatsuited for communication, and sends the converted data to thetransmission destination. When the communication unit 501 receives ane-mail, it converts the received mail data into a format suited fordisplay, and displays the converted data on the display unit 507.

Next, the image shooting function in the cellular phone 500 will beexplained. When the user operates the operation unit 510 to set theshooting mode and then instructs shooting of a still image or movingimage, the image sensing unit 505 performs shooting and sends the shotstill image data or moving image data to the image processing unit 506.The image processing unit 506 processes the shot still image data ormoving image data, and stores the processed data in the nonvolatilememory 508. The image processing unit 506 sends the obtained still imagedata or moving image data to the card I/F 511. The card I/F 511 storesthe still image data or moving image data in the memory card 512.

The cellular phone 500 can transmit, as a file attached to an e-mail, afile including still image data or moving image data shot in thismanner. More specifically, when transmitting an e-mail, an image filestored in the nonvolatile memory 508 or memory card 512 is selected, andtransmission of the image file as an attached file is instructed.

The cellular phone 500 can also transmit a file including shot stillimage data or moving image data to an external device such as a PC oranother phone via the external I/F 513. The user selects an image filestored in the nonvolatile memory 508 or memory card 512 and instructstransmission, by operating the operation unit 510. The control unit 509controls to read out the selected image file from the nonvolatile memory508 or memory card 512, and controls the external I/F 513 to transmitthe readout image file to the external device.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2013-254492, filed on Dec. 9, 2013 which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image capturing apparatus, comprising: animager including a plurality of pixels, each of the pixels including: aphotoelectric converter; a plurality of charge accumulation portionswhich accumulates a charge generated in the photoelectric converter; acontroller which controls accumulation of charge in each of theplurality of charge accumulation portions; and an output portion whichoutputs a signal corresponding to the charge from each of the pluralityof charge accumulation portions; and a generator which generates asignal of each of the pixels from the signals respectively output fromthe plurality of charge accumulation portions, wherein the photoelectricconverter is formed in a first chip, and the plurality of chargeaccumulation portions, the controller, and the output portion are formedin a second chip, and wherein the plurality of charge accumulationportions are arranged in parallel.
 2. The image capturing apparatusaccording to claim 1, wherein the controller causes a period of time foraccumulating a charge in the plurality of charge accumulation portionsto be different for each of the plurality of charge accumulationportions, and the generator selects, from among the signals respectivelyoutput from the plurality of charge accumulation portions, a signal of alarger level that does not reach a saturation level, and controller usesthe selected signal as a pixel signal of a corresponding pixel.
 3. Theimage capturing apparatus according to claim 1, wherein the controllerperforms control such that the period of time for accumulating a chargein the plurality of charge accumulation portions is the same, and thegenerator adds signals respectively output from the plurality of chargeaccumulation portions, and if an added signal resulting from theaddition does not reach a saturation level, selects the added signal,and if the added signal reaches the saturation level, selects one of thesignals respectively output from the plurality of charge accumulationportions and uses the selected signal as a pixel signal of acorresponding pixel.
 4. The image capturing apparatus according to claim1, wherein the controller performs control such that the period of timefor accumulating a charge in the plurality of charge accumulationportions is the same, the output portion outputs signals obtained bymultiplying the signals output from the plurality of charge accumulationportions by gains that are different from one other, and the generatorselects, from among the signals obtained by multiplying the differentgains, a signal of a larger level that does not reach a saturationlevel, and uses the selected signal as a pixel signal of a correspondingpixel.
 5. The image capturing apparatus according to claim 1, whereinthe controller causes timing of accumulating a charge in the pluralityof charge accumulation portions to be different for each of theplurality of charge accumulation portions, and the generator adds thesignals respectively output from the plurality of charge accumulationportions, and if an added signal resulting from the addition does notreach a saturation level, selects the added signal, and if the addedsignal reaches the saturation level, selects one of the signalsrespectively output from the plurality of charge accumulation portionsand uses the selected signal as a pixel signal of a corresponding pixel.6. The image capturing apparatus according to claim 5, wherein, when oneof the signals respectively output from the plurality of chargeaccumulation portions is selected, the generator selects a signal outputfrom the charge accumulation portion for which accumulation of charge isperformed at timing close to the middle of a period during which acharge is accumulated in the plurality of charge accumulation portions.7. The image capturing apparatus according to claim 5, wherein, when oneof the signals respectively output from the plurality of chargeaccumulation portions is selected, the generator selects a signal outputfrom the charge accumulation portion for which accumulation of charge isperformed at timing close to the start of a period during which a chargeis accumulated in the plurality of charge accumulation portions.
 8. Theimage capturing apparatus according to claim 5 further comprising acamera shake detection unit, wherein, when one of the signalsrespectively output from the plurality of charge accumulation portionsis selected, the generator selects a signal output from the chargeaccumulation portion for which accumulation of charge is performed attiming at which there is less influence of camera shake.
 9. The imagecapturing apparatus according to claim 1, wherein the first chip and thesecond chip are stacked on each other.
 10. The image capturing apparatusaccording to claim 9 further comprising an amplification unit configuredto amplify the signal of each of the pixels generated by the generator,according to a condition under which the signal was obtained.
 11. Theimage capturing apparatus according to claim 1, wherein the generatorgenerates signals which have a wide dynamic range from signalsrespectively output from the plurality of charge accumulation portionsas a pixel signal of a corresponding pixel.
 12. The image sensing deviceaccording to claim 1, wherein the first chip and the second chip arestacked on each other.
 13. An image capturing apparatus comprising: animager including a plurality of pixels, each of the pixels including: aphotoelectric converter; a plurality of charge accumulation portionswhich accumulates a charge generated in the photoelectric converter; acharge holding portion which holds the charges transferred from theplurality of charge accumulation portions; a controller which controlsaccumulation of charge in each of the plurality of charge accumulationportions; a transfer portion which transfers the charges from theplurality of charge accumulation portions to the charge holding portion;and an output portion which outputs each of signals corresponding to thecharges held in the charge holding portion; and a generator whichgenerates a signal of each of the pixels from the signals output fromthe charge holding portion, wherein the photoelectric converter isformed in a first chip, and the plurality of charge accumulationportions, the charge holding portion, the controller, the transferportion, and the output portion are formed in a second chip, and whereinthe plurality of charge accumulation portions are arranged in parallel.14. The image capturing apparatus according to claim 13, wherein theimager performs: non-additional output in which the charge accumulatedin one of the plurality of charge accumulation portions is transferredto the charge holding portion by the transfer portion, and the signalcorresponding to the transferred charge is output from the chargeholding portion by the output portion; and additional output in which,after the signal is output by the non-additional output, the chargeaccumulated in another one of the plurality of charge accumulationportions is further transferred to the charge holding portion by thetransfer portion and is added, and the signal corresponding to an addedcharge resulting from the addition is output from the charge holdingportion by the output portion, and, if a signal obtained by theadditional output does not reach a saturation level, the generatorselects the signal obtained by the additional output, and, if the signalobtained by the additional output reaches the saturation level, thegenerator selects the signal obtained by the non-additional output anduses the selected signal as a pixel signal of a corresponding pixel. 15.The image capturing apparatus according to claim 13, wherein the imagerreads out a signal for each frame in a predetermined cycle and performs:non-additional output in which the charge accumulated in one of theplurality of charge accumulation portions is transferred to the chargeholding portion by the transfer portion, and the signal corresponding tothe transferred charge is output from the charge holding portion by theoutput portion; and additional output in which the charges accumulatedin the plurality of charge accumulation portions are transferred to thecharge holding portion by the transfer portion and added, and the signalcorresponding to an added charge resulting from the addition is outputfrom the charge holding portion by the output portion, and thenon-additional output and the additional output are switched for eachrow or each frame according to a signal level of a previous frame. 16.The image capturing apparatus according to claim 13, wherein the firstchip and the second chip are stacked on each other.
 17. The imagecapturing apparatus according to claim 16 further comprising anamplification unit configured to amplify the signal of each of thepixels generated by the generator, according to a condition under whichthe signal is obtained.
 18. The image capturing apparatus according toclaim 13, wherein the generator generates signals which have a widedynamic range from signals respectively output from the plurality ofcharge accumulation portions as a pixel signal of a corresponding pixel.19. A method for controlling an image capturing apparatus including: animager including a plurality of pixels, each of the pixels including: aphotoelectric converter and a plurality of charge accumulation portionswhich accumulates a charge generated in the photoelectric converter, acontroller and an output portion, wherein the photoelectric converter isformed in a first chip, and the plurality of charge accumulationportions, the controller, and the output portion are formed in a secondchip, and the plurality of charge accumulation portions are arranged inparallel, the method comprising: a charge accumulation step ofcontrolling, by the controller, accumulation of charge in each of theplurality of charge accumulation portions; an output step of outputting,by the output portion, a signal corresponding to the charge from each ofthe plurality of charge accumulation portions; and a generation step ofgenerating, by a generator, a signal of each of the pixels from thesignals respectively output from the plurality of charge accumulationportions in the output step.
 20. The method for controlling an imagecapturing apparatus according to claim 19, wherein, in the chargeaccumulation step, a charge is accumulated for a different period oftime for each of the plurality of charge accumulation portions, and, inthe generation step, from among the signals respectively output from theplurality of charge accumulation portions, a signal of a larger levelthat does not reach a saturation level is selected and used as a pixelsignal of a corresponding pixel.
 21. The method for controlling an imagecapturing apparatus according to claim 19, wherein, in the chargeaccumulation step, a charge is accumulated in the plurality of chargeaccumulation portions for the same period of time, and, in thegeneration step, signals respectively output from the plurality ofcharge accumulation portions are added, and if an added signal resultingfrom the addition does not reach a saturation level, the added signal isselected, and if the added signal reaches the saturation level, one ofthe signals respectively output from the plurality of chargeaccumulation portions is selected, and used as a pixel signal of acorresponding pixel.
 22. The method for controlling an image capturingapparatus according to claim 19, wherein, in the charge accumulationstep, a charge is accumulated in the plurality of charge accumulationportions for the same period of time, in the output step, signalsobtained by multiplying the signals output from the plurality of chargeaccumulation portions by gains that are different from one other areoutput, and, in the generation step, from among the signals obtained bymultiplying the different gains, a signal of a larger level that doesnot reach a saturation level is selected and used as a pixel signal of acorresponding pixel.
 23. The method for controlling an image capturingapparatus according to claim 19, wherein, in the charge accumulationstep, a charge is accumulated at different timing for each of theplurality of charge accumulation portions, and, in the generation step,the signals respectively output from the plurality of chargeaccumulation portions are added, and if an added signal resulting fromthe addition does not reach a saturation level, the added signal isselected, and if the added signal reaches the saturation level, one ofthe signals respectively output from the plurality of chargeaccumulation portions is selected, and used as a pixel signal of acorresponding pixel.
 24. The method for controlling an image capturingapparatus according to claim 23, wherein, in the generation step, whenone of the signals respectively output from the plurality of chargeaccumulation portions is selected, a signal output from the chargeaccumulation portion for which accumulation of charge is performed attiming close to the middle of a period during which a charge isaccumulated in the plurality of charge accumulation portions isselected.
 25. The method for controlling an image capturing apparatusaccording to claim 23, wherein, in the generation step, when one of thesignals respectively output from the plurality of charge accumulationportions is selected, the signal output from the charge accumulationportion for which accumulation of charge is performed at timing close tostart of a period during which a charge is accumulated in the pluralityof charge accumulation portions is selected.
 26. The method forcontrolling an image capturing apparatus according to claim 23 furthercomprising a step of detecting camera shake by a camera shake detectionunit, wherein, in the generation step, when one of the signalsrespectively output from the plurality of charge accumulation portionsis selected, a signal output from the charge accumulation portion forwhich accumulation of charge is performed at timing at which there isless influence of camera shake is selected.
 27. The method forcontrolling an image capturing apparatus according to claim 19, whereinthe first chip and the second chip are stacked on each other.
 28. Themethod for controlling an image capturing apparatus according to claim19, further comprising a step of amplifying, by an amplification unit,the signal of each of the pixels generated by the generator, accordingto a condition under which the signal is obtained.
 29. The method forcontrolling an image capturing apparatus according to claim 19, wherein,in the generation step, signals which have a wide dynamic range aregenerated from signals respectively output from the plurality of chargeaccumulation portions as a pixel signal of a corresponding pixel.
 30. Amethod for controlling an image capturing apparatus including: an imagerincluding a plurality of pixels, each of the pixels including: aphotoelectric converter; a plurality of charge accumulation portionswhich accumulates a charge generated in the photoelectric converter; acharge holding portion which holds charges transferred from theplurality of charge accumulation portions, a controller, a transferportion, and an output portion wherein the photoelectric converter isformed in a first chip, and the plurality of charge accumulationportions, the charge holding portion, the controller, the transferportion, and the output portion are formed in a second chip, and theplurality of charge accumulation portions are arranged in parallel, themethod comprising: a control step of controlling, by the controller,accumulation of charge in each of the plurality of charge accumulationportions; a transfer step of transferring, by the transfer portion, thecharges from the plurality of charge accumulation portions to the chargeholding portion; an output step of outputting, by the output portion,each of signals corresponding to the charges held in the charge holdingportion; and a generation step of generating, by a generator, a signalof each of the pixels from the signals output from the charge holdingportion.
 31. The method for controlling an image capturing apparatusaccording to claim 30, wherein the imager is controlled bynon-additional output or additional output, in the non-additionaloutput, in the transfer step, the charge accumulated in one of theplurality of charge accumulation portions is transferred to the chargeholding portion, and, in the output step, the signal corresponding tothe transferred charge is output from the charge holding portion, and,in the additional output, in the transfer step, after the signal isoutput in the non-additional output, the charge accumulated in anotherone of the plurality of charge accumulation portions is furthertransferred to the charge holding portion and is added, and, in theoutput step, the signal corresponding to an added charge resulting fromthe addition is output from the charge holding portion, and, in thegeneration step, if the signal obtained in the additional output doesnot reach a saturation level, the signal obtained in the additionaloutput is selected, and if the signal obtained in the additional outputreaches the saturation level, the signal obtained in the non-additionaloutput is selected, and used as a pixel signal of a corresponding pixel.32. The method for controlling an image capturing apparatus according toclaim 30, wherein the imager reads out a signal for each frame in apredetermined cycle, and controls the imager by non-additional output oradditional output, and in the non-additional output, in the transferstep, the charge accumulated in one of the plurality of chargeaccumulation portions is transferred to the charge holding portion, and,in the output step, the signal corresponding to the transferred chargeis output from the charge holding portion, and, in the additionaloutput, in the transfer step, the charges accumulated in the pluralityof charge accumulation portions are transferred to the charge holdingportion and added, and in the output step, the signal corresponding toan added charge resulting from the addition is output from the chargeholding portion, and the method further comprises a switching step ofswitching between the non-additional output and the additional outputfor each row or each frame according to a signal level of a previousframe.
 33. The method for controlling an image capturing apparatusaccording to claim 30, wherein the first chip and the second chip arestacked on each other.
 34. The method for controlling an image capturingapparatus according to claim 30, further comprising a step ofamplifying, by an amplification unit, the signal of each of the pixelsgenerated by the generator, according to a condition under which thesignal is obtained.
 35. An image sensing device comprising: a pluralityof pixels, each of the pixels including: a photoelectric converter; aplurality of charge accumulation portions which accumulates a chargegenerated in the photoelectric converter; and an output portion whichoutputs a signal corresponding to the charge from each of the pluralityof charge accumulation portions, wherein the photoelectric converter isformed in a first chip, and the plurality of charge accumulationportions and the output portion are formed in a second chip, and whereinthe plurality of charge accumulation portions are arranged in parallel.